Art.-Schultz (F)

Current theories view learning as the acquisition of specific predictions1–4. Humans and animals learn to predict the outcomes of their behavior, including rewards. Learning depends on the extent to which these outcomes are different than predicted, being governed by the discrepancy or ‘error’ between outcome and prediction. Outcomes that affect learning in this way are termed ‘reinforcers’. Learning proceeds when outcomes occur that are not fully predicted, then slows down as outcomes become increasingly predicted and ends when outcomes are fully predicted. By contrast, behavior undergoes extinction when a predicted outcome fails to occur. (In the laboratory, predictions may fail either because the subject made an error or because the experimenter withholds the reward for correct behavior.) Recent learning algorithms employ errors in the prediction of outcome as teaching signals for changing synaptic weights in neuronal networks5. In these models, an unpredicted outcome leads to a positive signal, a predicted outcome to zero signal and the absence of a predicted outcome to a negative signal. The most efficient models capitalize on the observation that a key component of predictions concerns the exact time of reinforcement6,7. Their teaching signals use errors in the temporal prediction of reinforcement and compute the prediction error over consecutive time steps in individual trials (‘temporal difference’ algorithm8). Thus, teaching signals come to report progressively earlier reinforcement-related events and thus predict the outcome rather than simply reporting that it has occurred. They are particularly efficient for learning, as they can influence the behavioral reaction before it is executed. Reinforcement models that use predictive teaching signals can learn a wide variety of behavioral tasks, from balancing a pole on a cart wheel9 to playing world-class backgammon10. It is therefore of interest to determine whether real nervous systems might process rewards in a similar manner during learning. Results from lesioning and psychopharmacological experiments indicate a role of dopamine systems in behavior driven by rewards and in reward-based learning12–14. We have studied the neural mechanisms underlying this role of dopamine in monkeys and have previously reported that midbrain dopamine neurons show responses to food and liquid rewards that depend on their predictability15,16. The present study investigated whether these responses could have the formal characteristics of teaching signals. We found that the magnitude of dopamine responses to a juice reward reflected the degree of reward predictability during individual learning episodes. An unexpected reward evoked a strong response in dopamine neurons. As the monkeys’ performance improved (i.e. as they learned to predict which response would trigger a reward), the neuronal response to the reward progressively decreased. Moreover, by varying the timing of reward, we found that dopamine neurons signal not only its occurrence but also its timing relative to expectations. Thus dopamine neurons seem to track the reward prediction error and emit a signal that has all the typical characteristics of a positive reinforcing signal for learning.